Method for Agglomerating Onscreen Objects 
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BACKGROUND OF THE INVENTION 
Field of the Invention 

[0004] This invention relates to methods and devices for entering user inputs 
into an electronic device, and, more particularly, to machines such as computers 
20 that include a screen display and graphic input means for the user. 

Description of Related Art 

[0005] The copending patent applications captioned above describe a 
graphical user interface for a machine having a screen display. A significant 
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feature of the interface is that it provides the means for hand drawn entry of on- 
screen objects which may be associated with functions, files, connections, and 
other objects or actions accessible by the system to carry out the user's desired 
purpose, whatever it may be. A fundamental aspect of this interface is the ability 

5 to carry out commands and actions corresponding to the hand drawn inputs as 
they are applied to any on-screen object, including (but not limited to) objects 
representing files (text, data, sound, video, graphics, photos, and the like), 
functional devices (audio processing, video processing, graphic and photo 
processing, text editing, data processing, internet communications, and the like), 

10 and interconnecting arrows and lines that link these files and functional devices 
in on-screen arrangements that are drawn by the user to accomplish desired tasks. 
The prior related applications also introduce arrow logics as a method for 
inputting transactions involving objects displayed onscreen through the use of 
arrows and lines drawn between such objects. 

15 
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BRIEF SUMMARY OF THE INVENTION 

[0006] The present invention generally comprises a method for user 
customization of a graphic user interface for an electronic device that includes a 
screen display. More specifically, in a computer-user interface that provides 

5 recognition of hand-drawn inputs, and assignment of the inputs to functions, files, 
connections, and other objects or actions accessible by the system to carry out 
the user's desired purpose, the invention provides a computer system in which 
one onscreen object may be combined with a second onscreen object, and one of 
the onscreen objects may become invisible in the transaction. Thus, for example, 

10 an onscreen object such as a touch-actuated switch may be dragged (or directed 
by appropriate arrow logic inputs) to overlay another onscreen object, such as a 
graphic object like a photo. The switch may become invisible, while retaining its 
functionality, and the photo image will be displayed in conjunction with the 
switch position, whereby the switch function is effectively combined with the 

15 photo image. 

[0007] Likewise, the method may include directing or dragging an object, such 
as graphic object, to be superimposed on a functional object such as a switch or 
the like, the graphic object substantially or completely obscuring the view of the 
underlying functional object. The overlaying object is made touch-transparent, 
20 so that any click or touch onto the graphic object is transmitted to the functional 
object underneath, whereby the functionality of the underlying object is 
effectively combined with the overlying object. 

[0008] The method is not limited to the use of graphic objects or functional 
objects, but may be used to combine any two onscreen objects where the 
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combination creates a beneficial joining of characteristics of one object with 
another. Without the invention, software writers would have to create code to 
recognize every user customizable item that is desired to be used in the 
assignment process, resulting in an unacceptably large amount of code. This 

5 invention enables software writers to use a much smaller software package to 
recognize a small group of objects, and provide virtually inf inite customization by 
the user through user-directed combination of the machine recognizable objects 
with a large number of onscreen objects. Thus, for example an onscreen button 
that commands an instant message to be sent to a particular individual may be 

10 melded with a photo image of that individual. The button may be made invisible 
and the button function is accessed by recourse to the photo image, so that the 
user may click or tap on the photo to direct an instant message to that person. 
[0009] The method of the invention may include a further step of "gluing" 
together the two superimposed objects, whereby the glued objects are joined as if 

15 mechanically bonded; e.g., for movement by the user on the screen, for arrow 

logic commands, and the like. This gluing step is made explicit through the use of 
an info canvas that elicits a response from the user; i.e., yes or no to gluing the 
two objects. This gluing step eliminates accidentally combining two onscreen 
objects by unintentionally overlapping or intersecting them. 

20 [0010] In general, the invention deals with two broad classes of onscreen 

objects: 1) those that have received an assignment, such as a function (i.e. switch, 
fader) or action (i.e., open a file, display a text portion), and, 2) those that have not 
received an assignment. The combination of an object that has an assignment 
with an object that has no assignment, or another object that has an assignment, 
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may be carried out through explicit steps (described below), or may be 
accomplished merely by the context in which the two objects are combined. The 
following contexts are illustrative, but not limiting examples: 
[0011] 1 . Dragging a first object over a second object having an assignment 
5 partially or completely obscures the second object, and automatically causes the 
first object to be touch transparent so that a click or tap applied to the first object 
is transmitted to the second, underlying object. 

[0012] 2. Simply gluing an assigned-to object to any graphic object, picture 
object, or video object results in the top object becoming touch transparent with 

10 respect to the bottom object. 

[0013] 3. Dragging an assigned-to object over a graphic object causes the 
assigned-to object to become invisible, but still functional, so that the graphic 
object is visualized and the functional object may be actuated by touch or click. 
[0014] 4. Dragging a graphic or photo object over a switch object, so that the 

15 perimeter of the switch remains visible, causes the overlying object to remain 
visible but become touch transparent. When the switch or overlying object is 
touched or clicked, the switch changes state and alters its color or brightness to 
indicate actuation. 

[0015] 5. Dragging one assigned-to object over another assigned-to object 
20 creates a combined onscreen toggle object, whereby clicking once on the 

combined object actuates the first object and clicking again actuates the second 
object. 

[0016] In this patent application the terms gluing, agglomerating, melding, 
associating, and bonding are synonyms that indicate the functionality and actions 
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of a first onscreen object are joined to a second onscreen object, while the 
appearance of the joined objects is substantially the same as one of the two 
objects. 

[0017] The method of the invention is not limited to dragging one object to 
5 overlap or intersect another object, or employing arrow logics to direct the 
combination of two object. Also, the system may provide a menu selection in 
which two selected objects are combined to include the traits of both objects, 
such as the appearance of one object and the functionality of the other object. 
Likewise, the combination may be elicited by use of an icon, a command 
10 statement, verbal command or commands, or any other computer input technique 
known in the art. 
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BRIEF DESCRIPTION OF THE DRAWING 

[0018] Figures 1 A-1D are a sequence of screen displays depicting the steps of 
the method of the invention for combining two onscreen objects. 

5 [0019] Figures 2A-2C are a sequence of screen displays depicting a further 
example of the steps of the method of the invention for combining two onscreen 
objects. 

[0020] Figure 3 is a depiction of an Info Canvas displayed in accordance with 
10 the method of the invention. 

[0021] Figures 4A-4C are a sequence of screen displays depicting another 
example of the steps of the method of the invention for combining two onscreen 
objects. 

15 

[0022] Figure 5 is a depiction of an Info Canvas displayed in accordance with 
the method of the invention. 

[0023] Figures 6A-6D are a sequence of screen displays depicting a further 
20 example of the method of the invention for combining two onscreen objects. 
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DETAILED DESCRIPTION OF THE INVENTION 
[0024] The present invention generally comprises a method for user 
customization of a graphic user interface for an electronic device that includes a 
screen display. More specifically, the invention provides a system in which one 
5 onscreen object may be combined with a second onscreen object, and one of the 
onscreen objects may become invisible or touch transparent in the transaction. 
The user may interact with the visible onscreen object to access the invisible 
object. 

[0025] With regard to Figures 1A-1E, there is shown a sequence of views of 
10 onscreen objects on a computer display, and the user inputs required to carry out 
one embodiment of the method of the invention. In Figure 1 A, the display is 
exhibiting a switch 21 , which is an onscreen object to which an assignment has 
been made, so that tapping or clicking or otherwise actuating the switch 21 
causes a defined action to occur within the computer system. For example, the 
15 action may comprise calling forth a user-defined file such as a photo, audio, video, 
or text; establishing a network connection or email update; beginning playback 
of an audio or video file, or carrying out any user-defined or default transaction. 
[0026] The user may wish to replace the appearance of the switch with a 
graphic that is more attractive, or which represents more explicitly the function of 
20 the switch 21 . To create a star graphic element, the user draws a star 22, which is 
recognized by the software system and replaced by a star 23 as shown in Figure 
IB. The user may then cause the switch 21 and star 23 to overlap or intersect or 
otherwise be combined. As shown in Figure IB the user may click and drag on 
the star 23 to translate it to be superimposed on the switch 21, or may employ an 
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appropriate arrow logic input, as shown in Figure 1C to cause the same effect. In 
either case, the result shown in Figure ID is that the star 23 overlays the switch 
21. The mere superposition of the two onscreen objects may be a sufficient 
action to cause the two objects to be glued together and combined as described 

5 below, based on the context of the action. As an alternative, however, a double 
tap/click or right click on the switch 21 elicits an Info Canvas 24 which displays 
significant data about the switch 21 . Included in the Info Canvas is a user- 
selected item "Glue", which indicates the user may select this item to cause the 
switch 21 to be glued or agglomerated with the red star 23. Likewise, the user 

10 may select "Make touch transparent" so that any click or tap imparted to the star 
23 is passed through to the underlying switch 21. (Note that the Info Canvas for 
the red star 23 likewise could have been accessed and used to glue the two 
objects together.) Thus the agglomerated objects combine the function of the 
switch 21 with the appearance of the graphic object 23, thereby enabling 

15 customization of the screen display by the user. 

[0027] Another example of the method of the invention involves an onscreen 
object that accepts assignments, and an onscreen object that is a photo file. With 
regard to Figure 2A, an onscreen object such as red switch 31 may be drawn or 
otherwise caused to be displayed. The red switch 31 may be assigned the action 

20 of checking email whenever the switch 3 1 is actuated by a click or tap or the like. 
The user may bring onto the display a photo 32 of an old mailbox, and bring the 
two objects 31 and 32 together to be superimposed or intersecting by dragging 
the switch 3 1 over the mailbox picture, as shown in Figure 2B, or by the use of 
arrow logic transaction, or the equivalent. The context of this action may 
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accomplish gluing the two objects together, as described previously, or the Info 
Canvas 34 (Figure 3) of either object 31 or 32 may be accessed to carry out the 
gluing or agglomerating step, and the step of making the switch 31 invisible. 
Thereafter the switch 31 becomes invisible, and the agglomerated object 33 of 
5 Figure 2C has the appearance of the photo 32 (a mailbox), while the functionality 
of the red switch 3 1 is maintained. Thus, the user may tap or click on the mailbox 
33 to carry out the task of checking email. 

[0028] Note that in general the manner in which an onscreen object (such as a 
switch) is made invisible (or touch transparent) may be carried out in any of 
10 several ways, such as a verbal command, a typed command, selecting an entry in a 
menu, drawing an arrow or other graphic object that initiates the action of making 
the object invisible or touch transparent, selecting both objects (as with a lasso) 
and gluing them together, and the like. 

[0029] With regard to Figures 4A-4C, another example of a way to carry out 
15 the method of the invention involves bringing onscreen an image 32' of a 

mailbox, and a switch 31' labeled "Check email" (Figure 4A). The user may drag 
or otherwise cause the image 32' to be superposed over the switch 31\ as shown 
in Figure 4B, whereby the switch 3 V is completely obscured. Generally 
speaking, in most operating systems an unseen onscreen object cannot be clicked, 
20 tapped, or activated unless it is made visible. However, as shown in Figure 5, the 
user may call forth an Info Canvas for the image by double clicking/tapping or 
right clicking on the image 32'. The Info Canvas 37 provides a entry "Touch 
Transparent ON," which, when selected, causes any touch or click applied 
thereafter to the image 33' to be transferred to the underlying, unseen switch 31'. 
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Thus once again two onscreen objects have been combined, and the combination 
provides the appearance of one object with the functionality of the other object. 
[0030] With reference to Figures 6A-6D, a further example of the method of 
the invention involves placing onscreen a first assigned-to object 41, such as 

5 switch 41 . The assignment of the switch 41 is a calendar 42, which may contain 
personal date entries and the like. Clicking or tapping on switch 41 causes the 
calendar 42 to appear onscreen, and a subsequent click or tap causes the text to 
disappear. A second assigned-to object 43 may be placed onscreen (Figure 6B), 
the object 43 being a switch that directs the display of an address book 44. With 

10 regard to Figure 6C, the switch 41 may be agglomerated with the switch 43, by 
clicking and dragging, arrow logic command, menu selection, or the like. Thus 
two assigned-to objects are combined, a situation that is unlike any of the 
previous examples. The combination may be made explicitly by recourse to the 
Info Canvas 46 and selection of the "Glue" entry, or may be made implicitly by 

15 the context of the action (one assigned-to object being dragged over another 
assigned-to object).The combined switches form a toggle switch 47 (Figure 6D). 
One click or tap on the toggle switch 47 calls forth onscreen the calendar 42, and 
the next click or tap on switch 47 causes the address book 44 to be displayed. 
[0031] It may be appreciated that the software system may be encoded to 

20 recognize a range of hand drawn objects, such as rectangles, circles, stars, letters, 
triangles, and the like. These hand drawn objects may accept assignment of a 
wide variety of transactions, becoming powerful tools that can be redrawn 
onscreen at any time to be recalled for immediate use. The invention permits a 
second onscreen object that has no assignment, or is not capable of accepting 
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assignment, to be agglomerated to an assigned-to object, so that the functionality 
of the assignment is coupled to the appearance of the second object. This 
capability greatly increases the variety and range of customizable onscreen 
objects, without substantially increasing the size of the code required for this 

5 customizable screen environment. 

[0032] Note that any Info Canvas 24, 34, 37, or 46 that provides for an object 
to be glued to another will, when the glued state exists, provide a selection to 
"UnGlue" the two objects. Likewise, any Info Canvas that provides for an 
object to be made touch transparent or invisible will display the reverse entry, 

10 "Touch Transparent OFF" or "Make visible" to enable the user to cancel or 
reverse the action. 

[0033] The examples described above indicate that the invention deals with 
two broad classes of onscreen objects: 1) those that have received an assignment, 
such as a function (i.e. switch, fader) or action (i.e., open a file, display a text 

15 portion), and, 2) those that have not received an assignment. The combination of 
an object that has an assignment either with an object that has no assignment, or 
another object that has an assignment, may be carried out through explicit user 
action, or may be accomplished merely by the context in which the two objects 
are combined. The following contexts are illustrative, but not limiting examples: 

20 [0034] 1 . Dragging a first object over a second object having an assignment 
partially or completely obscures the second object, and automatically causes the 
first object to be touch transparent so that a click or tap applied to the first object 
is transmitted to the second, underlying object. 
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[0035] 2. Simply gluing an assigned-to object to any graphic object, picture 
object, or video object results in the top object becoming touch transparent with 
respect to the bottom object. 

[0036] 3. Dragging an assigned-to object over a graphic object causes the 
5 assigned-to object to become invisible, but still functional, so that the graphic 
object is visualized and the functional object may be actuated by touch or click. 
[0037] 4. Dragging a graphic or photo object over a switch object, so that the 
perimeter of the switch remains visible, causes the overlying object to remain 
visible but become touch transparent. When the switch or overlying object is 
10 touched or clicked, the switch changes state and alters its color or brightness to 
indicate actuation. 

[0038] 5. Dragging one assigned-to object over another assigned-to object 
creates a combined onscreen toggle object, whereby clicking once on the 
combined object actuates the first object and clicking again actuates the second 
15 object. 

[0039] This invention is adapted to be used with any electronic device that 
includes a screen display in which onscreen objects may be manipulated by a user 
to effect inputs into the electronic device. 

[0040] The foregoing description of the preferred embodiment of the invention 
20 has been presented for purposes of illustration and description. It is not intended 
to be exhaustive or to limit the invention to the precise form disclosed, and many 
modifications and variations are possible in light of the above teaching without 
deviating from the spirit and the scope of the invention. The embodiment 
described is selected to best explain the principles of the invention and its 
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practical application to thereby enable others skilled in the art to best utilize the 
invention in various embodiments and with various modifications as suited to the 
particular purpose contemplated. It is intended that the scope of the invention be 
defined by the claims appended hereto. 
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